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^ Abstract 

^2 In this paper we show that, under certain generic conditions, bihiards on ovals have only a 

^ finite number of periodic orbits, for each period A'^, all non-degenerate and at least one of them 

is hyperbolic. Moreover, the invariant curves of two hyperbolic points are transversal. We 

explore these properties to give some dynamical consequences specially about the dynamics 

jy^ in the instability regions. 

Q 
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1 Introduction 



' — ' Let a be an oval, i.e., a planar, closed, regular, simple, oriented counterclockwise, C' curve, / > 2, 

^ parameterized by t G S*^ and such that its radius of curvature R{t) > 0. The billiard problem on 

a consists in the free motion of a point particle on the planar region enclosed by a, being reflected 

^ elastically at the impacts on the boundary. The trajectories will be polygonals on this planar 

CTn region. 
O 

1^ Since the motion is free inside the region, it is completely determined by the points of impacts at 
a, and the direction of motion immediately after each reflection, defined by the angle 6 between 
it and the oriented tangent to the boundary at the reflection point. Therefor, to each oval a is 

^ associated a billiard map Ta from the cylinder x (0, vr) into itself which to each initial condition 
{to,6o) associates the next impact and direction: Ta{tQ,6Q) = (^1,6*1). 

^ The map Tq, has some very well known properties (see for instance [2], [9], ^Q\, [E], [E]): It is a 
ci C'~^-diffeomorphism, preserving the measure dfi = ^ sm9 d9 dt, where s = s{t) is the arclength 
parameter for a. It is reversible with respect to the reversing symmetry H{t,6) = (t, vr — 0), 
which satisfies H o H = Id and H oT^ = T^^ o H. It is a monotone Twist map. So, the billiard 
map defines a discrete reversible conservative bidimensional dynamical system in the cylinder 
5*^ X (0, tt). This map has no fixed points but Birkhoff ^ proved that it has periodic orbits of any 
period greater or equal to 2. 

In this paper we show that, under certain generic conditions, billiards on ovals have only a finite 
number of periodic orbits, for each period A^, all non-degenerate and at least one of them is 
hyperbolic. Moreover, the invariant curves of two hyperbolic points are transversal. The generic 
existence, in the analytic case, of at least one nondegenerate periodic orbit for each period, was 
proved by Kozlov and Treschev [TU] . Our results generalize theirs in the sense that we work with 
curves at least C^, prove the finiteness and study the heteroclinic connections. 
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Once we have a hyperbolic periodic orbit, which unstable manifold project to S^, we can construct 
the instability region associated to it from the closure of the invariant manifold. The dynamical 
object we obtain is the same as the one constructed by Le Calvez in [13] and thus shares the same 
dynamical properties. 

2 Generic Periodic Orbits 

Billiard maps are a special kind of diffeomorphisms and perturbations of the map may produce 
diffeomorphisms which are not billiards. In order to assure that the perturbed diffeomorphism 
remains a billiard, we have to perturb the boundary curve instead of the map itself. 

As planar rigid motions and homotheties do not change the geometrical features of the curve, they 
do not change the dynamical characteristics of the associated billiard map and so it is useful to work 
on the set of curves modulo this equivalence relation. Let C be the set of equivalence classes of 
ovals. Given a representative a of an equivalence class [a] G C, the normal bundle {a(t),ri(t)) is C^. 
For e > 0, a tubular neighbourhood of a is given by N^{a) = {a(t)+Xri(t), < t < 1, —e < X < e}. 

Definition 1 [f]] E C is e- close to [a] E C if there exist representatives a in [a] and (3 in [[3] 
such that the image of (3 is in N^{a), and its canonical projection onto the image of a is a 
diffeomorphism. 

As a consequence, P can be written as P(t) = a{t) + \(t)ri(t), with A at least and periodic. 
Definition 2 G C is e-C'^ -close to [a] E C if is e-close to [a] and \ \X\\2 < e- 
Proposition 3 C is a Baire space. 

Proof: C^(S'\ R^) with the C^-topology is a Baire space. By Sard's Theorem, the subset I 
of immersions is open and dense in (7^(5"^, R^), and so also a Baire space. For a G / let R{t) 
be its radius of curvature on each t and let index be the total curvature divided by 2it. Let 
A = {a such that i? > and index = 1} and B = {a such that R > and index = 1}. Note 
that B is exactly the set of closed, regular, simple, oriented, C^-curves with positive radius of 
curvature. It is clear that A is closed and B is open and dense in A, so B, the set of ovals, is 
Baire with the C^-topology. Cutting by the equivalence relation and taking the induced topology, 
we have that C is a Baire space. ■ 

As we have showed in by perturbing the classes of curves we get nearby diffeomorphisms in 
the C^-topology: 

Proposition 4 // [P] E C is e-C^ -close to [a] G C then there exist representatives a in [a] and (3 
in [(3] such that the associated billiard maps and Tp are -close. 

To lighten the notation, in what follows we will denote both the curve a and the equivalence class 
[a] by a, unless where confusion may happen. 
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2.1 Periodic Orbits 



Let « G C. Since the radius of curvature R{t) is strictly positive, a can be reparameterized by 
if G [0, 27r), the positive oriented angle between the tangent vector a'(t) and a fixed direction (say, 
the X-axis). Let Tq, : [0, 27r) x (0, tt) [0, 27r) x (0,7r) be the associated billiard map. 

A point (^O)^o) £ [0, 27r) x (0,7r) is n-periodic if n is the smallest positive integer such that 
T^(^Q, ^o) = (^0) ^o)- Such a point defines an n-periodic orbit 

{(^0,^0),i;(^0,^0) = (^i,^l),...,Trl(^0,^0) = iWn-lA-l)}. 

An n-periodic orbit is nondegenerate if each of its points {^j,9j) is a non degenerate fixed point 
of T^. Otherwise it is degenerate. 

Let Un the set of ovals a E C such that for each n 1 divisor of N, the associated billiard map 
Tq, has only nondegenerate n-periodic orbits. 



Proposition 5 If a E Un then for each n ^ 1 divisor of N, Ta has only a finite number of 
n-periodic orbits. 

Proof: Let {(^q, ^o), (^i, ^i), (^n-i> ^n-i)} be an n-periodic orbit of Tq. ThenQ;(^Q), a{ip^),..., 
a{(pn-i) are the vertices of a polygon inscribed in a. Let f3i be its internal angles and 7^ — 9i if 
9i < n/2 or = n -9i if 9i > ti/2. As YJi=Q Pi < {n - 2)n, 2ji + /?j = tt and then Yli=o li > ^r. 
So there is at least one j such that vr/n < 9j < (n — l)7r/n. 

This means that if a G L^at then each n-periodic orbit of T^, with n 7^ 1 divisor of iV, has at least 
one point in the compact cyhnder [0, 27r) x [vr/iV, [N — l)7r/iV]. 

Since all the points of are nondegenerate, there is only a finite number of them on the compact 
cylinder and so, only a finite number of n-periodic orbits. ■ 

For diffeomorphisms on [0,27r) x (0,7r), with the C"^ -topology, having only a finite number of 
nondegenerate fixed points on a compact subset is an open property. Taking the restriction to 
billiard maps, we get 



Proposition 6 is open onC. 



By Birkhoff's Theorem, any a G C has periodic orbits of any period. So a will be outside Un 
if it has (finitely or infinitely many) degenerate n-periodic orbits for n ^ 1, divisor of N. The 
following lemma provides the basic tool for proving the density of Un- 

Lemma 7 Suppose that O = {(^q, 6'o), (^1, 6*1), (^„_i, ^^„_i)} is a degenerate n-periodic orbit 
for the billiard map Ta associated to a curve a & C. Then there is a curve a & C, C^-close to a, 
such that O is a nondegenerate n-periodic orbit for the associated billiard map Tq. 



Proof: Let Xj = R{ipj) sm9j and Ij = ||Q;(<^j) — Q;((^j+i)||. The Jacobian matrix of Tq at {ipj,9j) 
is given by: 



DTa{(pj,9j) 



Ij Xj Ij 



and, for any point {cpj^dj) of O, det (^DT^{'(pj, 9 j)) = 1. It follows that O is degenerate if 
tr [DT2{Tpp 9j)) = ±2, for any point of the orbit. 
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We have that 

tr {DT^{^o,eo)) = tr (^^(^^.i, ^„_i)...L>r,(^o, ^o)) = tr (^„_iA„_2...AiAo) , 

where 

/ 







h \ 


Xj 




Xj 


Xj+l ^ 


Xj+l 


Xj 


Xj 


^3 1 



Let us isolate the terms that depend on, for instance, x\. Only Ai and have entries with xi 
and 



Xi 

1 

Xi 



Xo xo ^ 



-X2 



h lo 

Xo 

h k) k) X2 



Xo 



Xo 



h+lo h+lo 

Xo Xo 

h+lo X2 h+lo X2_ 

Xo Xo Xo Xo 



Bi + Cifi 



Then 



tr(r>T„"(^o,^o)) = tr 



^n-l^n-2---^2 { — Bi +Ciq 
' Xi 



= — tr (A„_i^„_2...A2Si) + tr (A„_i>l„_2...A2Ci,o) 

Xi 

= h Ci 

Xi 

where neither bi nor ci depend on xi. 

If bi 7^ 0, let / be an interval such that G / and ^j ^ I for j ^ 1. Let q;(</7) = Q;((/7) + \{(p)r]{ip)^ 
where A is a periodic function satisfying A((/7) = if </? ^ /, A(^i) = A'(^i) = 0, A"(^i) 7^ 
and ||A||2 small enough to guarantee that i?, the radius of curvature of a, is strictly positive. 

The perturbed curve a and the original one a coincide, except on a neighbourhood of and, 
at they have a contact of order one. So the polygonal trajectory that corresponds, on the 
billiard table, to the periodic orbit is unchanged and O is also a n-pcriodic orbit for Tq,. 

Moreover Ij = \\a{ipj) — Q;(^j+i)|| = lj,^j, the radius of curvature R{^j) = R{}Pj) if j 7^ 1 and 
Then 

tr (Dim. m = = ^F^^ + = ^' - St) + -^^ 

and we can choose A as small as we want such that tr (^DT^(TpQ, Oq)) 7^ dz2 and O is a nondegenerate 
n-periodic orbit for T^. 

If 61 = then 

tr {DT^(^o,Oo)) = tr (^„_iA„_2...^2Ci,o) 

with 



A.2 Ci Q — B2 + C2 

X2 



where 6*2,0 has the same form of Aq, replacing xi by X3 and /q by I2 + h + 
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Then 



tr (Z}T:(^o,M = - + C2 

^2 



where neither 62 nor C2 depend on xi and X2- If 62 7^ then we can make the normal perturbation 
on a neighbourhood of as above. 

If 62 = we continue the process until finding a 6j 7^ and then making the normal perturbation 
at or to end up with all 6j's = 0, for i = l..n — 1 in which case, as xq = Xn, we will have 

tr(DT„"(^o,^o)) = tr(C„_i,o) 

l„-l + ... + ll+lo -| 'n-l+--- + 'l+'o 



^' <n-l + ... + h+f0 _ 2 ln~l + -+ll+lo _ 

Xo XQ 

^ ^^f ln-1 + + h _ ^ 

V Xq 

As In-i + . . . + /i + /q is the perimeter of the polygonal trajectory and then different from 0, we 
can perform the normal perturbation on a neighbourhood of Tp^ as above concluding the proof of 
the lemma. ■ 



Proposition 8 Un is dense on C. 



Proof: Given a ^ Un, let Va be the set of all fixed points of and H : [0, 27r) x (0, vr) ^ [0, 27r) 
be the projection on the first coordinate. 

As a ^ Un, there is a (j^^, 6q) G Va such that 0{Jpq, Oq) is a degenerate n-periodic orbit of Tq, 
for n ^ 1, divisor of A^. By lemma [t] we can find a curve ai, close to a, such that 0(^q,6'o) is 
a non-degenerate n-periodic orbit of T^^ and so there are intervals Jo, Ji, Jn-i C [0, 27r) such 
that n (T^^(^q, 9q)) is the unique point of UiVai) in Jj. 

As is compact, we can construct after a finite number of steps, a curve 02, as close as we want 
to a, such that 11(^0,2) = {ipojipi, ...yipk} and intervals j7o, J7i, Jfc C [0,2n), with UJ7^- = [0,27r), 
and such that each ipj is the unique point of 11(^02) in J'j and so each fiber U'^^ipj) contains at 
least one nondegenerate fixed point and maybe other degenerate or nondegenerate fixed points. 

If has only non-degenerate fixed points, the problem is solved. Otherwise, let {ipo, 6'o) and 
(V'c^o) be two different fixed points of T^, with {iPq,9q) non-degenerate and {i'o,C,o) ^ [0, 27r) x 
[tt/N, {N - l)n/N] degenerate. 

If {ipo, ^0) = Tl^{ipo, 9o) then they are both non-degenerate, so suppose that they do not belong to 
the same orbit. Let n and m be the periods of the orbits, respectively. Then tr {DT^^ {ipo, Oq)) 7^ ±2 
and tr (DT™(^o, ^o)) =±2. 

As in the proof of lemma [rj there is an / such that tr (Z}T^(^/'0) ■Co)) = ^ + with 6/ 7^ and 
then a normal perturbation in the interval containing ipi will produce a curve f3, C^-close to a, 
such that both of (ipoy^^o) and (V'o^^o) are nondegenerate fixed points of . 

To finish the proof of the proposition, we remark that [tc/N, {N — l)7r/A^] is a closed interval. So, 
after a finite number of perturbations we can construct a curve in Un, arbitrarily close to the 
given a ^ Un- • 
Taking the intersection of the open and dense sets Uns and remembering that C is a Baire space 
it follows that 

Theorem 1 Generically, for billiards on ovals, there is only a finite number of periodic orbits, 
for each period N, and they are all non-degenerate. 
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2.2 Intersection of stable and unstable manifolds 

Let a be aC' oval, the associated billiard map and F„ : R x (0, tt) i— >• R x (0, vr) a lift of Tq,. 
If C(^0) ^o) = {(^05 ^0)5 (jPi, On-i)} is an n-periodic orbit of Tq, then there is an m, 

1 <m < n such that F^(J^q, 9q) = (Jpq + 2m7i, 9q). We will say that 0(Jpq, 6q) is an (m, n)-periodic 
orbit. 

Given n and m, 1 < m < n, let 

Gm,„(V^o, V'l, , •••,^n-i) = -\\a{i)o) - - ||a(^i) - a(V^2)|| - ••• - ||a(^n-i) - "(V'o + 2mn)\\. 

0(Jpq,9o) = {(jfQ,9o),(jpi,9i),...,(jPn^i,9n~i)} is a non-degenerate (m, n)-periodic orbit if and 
only if (^05^1) • • -5 ^n-i) is a non-degenerate singularity of Gm,n- By the Mackay-Meiss criterion 
|15j . non-degenerate minima of Gm,n correspond to hyperbolic orbits and non- degenerate maxima 
to elliptic or reverse hyperbolic ones. 

li a & Un then all the n-periodic orbits, n 7^ 1 divisor of A^, are nondegenerate singularities for 
the appropriate Gm,n and then are hyperbolic or elliptic. Actually, for each n and each m such 
that gcd(m, = 1, the first step in the proof of Birkhoff's Theorem is the existence of a global 
minimum of Gm.n- So there is at least one hyperbolic ra-periodic for T^. The stability of the 
other periodic orbits is strongly related to the geometry of the oval a, and they can even be all 
hyperbolic, like the 2-periodic orbits in the examples given in [1] or [TTj. 

Let {(^0; ^0)) (^1) ^1); (^n-i5 ^n-i)} bc ouc n-pcriodic hyperbolic orbit. Associated to each point 
{ipj,9j) there are two C'~^ invariant curves 

J^.) = {{^,9) such that TJ'{^,9) {■^j,9j) as i -00} 

and 

W'{ipj,9j) = {{ip,9) such that TJ'{ip,9) {ipj,9j) as z ^ 00} 
called, respectively, unstable and stable curves of (jpj,9j). 

A point {(po,9o) is hetero(homo)chnic if {(po,9o) G W'^{'(pj,9j) fl W^{'(pi^,9k) for k ^ j {k = j). 
For the hyperbolic periodic orbits corresponding to global minima of Gm,ni Bangert's results pjj 
assures the existence of hetero and homoclinic points. Other hyperbolic periodic orbits may have 
only homoclinic points, as the 2-periodic orbit plotted in the figure bellow. 

We don't know if, for a generic oval a there exists a hyperbolic periodic orbit without hetero 
or homoclinic points. In particular, W^{Tpj,9j) fl 6*^) = is an open property for C^- 

diffeomorphisms, and so will be open for billiards on ovals. 

An heteroclinic or homoclinic point (yjo, 9q) is called transversal (tangent) if the invariant stable 
and unstable curves meet transversally (tangentially) at ^o)- Transversal intersection of stable 
and unstable curves is also an open property for C^-diffeomorphisms, and so it will be open for bil- 
liards on ovals. In the next lemma we will prove that any billiard with a tangent hetero (homo) clinic 
point can be approached by billiards with a transversal one. We use the techniques introduced in 
[H] and used in [5]. 

Lemma 9 Let a G Un and {(^q,6'o), (^„_i, ^^„_i)} he an n-periodic hyperbolic orbit such that 
a stable and an unstable curve, say W'^(JPq,9o) and W^(j^j,9j) intersect tangentially at {{Pq,9q). 
Then a can be approximated by curves in such that {(Pq,9q) is a transverse heteroclinic (or 
homoclinic) point of the associated billiard map. 
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Proof: Since (v^c^o) G {W'(jfo,9Q)nW''(j^j,9j)) there are sequences {ipi,9i) = T^"(v9o,6'o) ^ 
(^O)^o) and {{p^i,9^i) = T^''"'{ipQ,9o) — > (Tpj,9j) as z ^ +00. Then, there exists an interval /i 
such that (fo ^ h, '^o,'^i, ■■■,'^n ^ -^1 'fm ^ /i, Vm G 

As is a C'~^-diffeomorphism, each 6'j) is also a heteroclinic tangent point for every i. 
Moreover, as has the Twist property, it is not possible to the tangency at every {ipin, 9in) to 
be vertical. So we can suppose that the stable and unstable curves are local graphs over the 
93-axis at a neighbourhood of, for instance, (y9o,6'o). So there is an interval I2 containing ipo such 
that W^{Tpq,9q) and W^{^p9j) are given locally by the graphs of ^ = 9^{ip) and 9 = 9'^{(p), with 
P(y,o) = ^"(¥^0) = 9o and f ((^0) = ^(^0) = 9',. 

Those graphs define two pencils of rays that focuses forward and backward at the distances (see, 
for instance [T6] ) 

i?osin6'o , , i?osin6'o 

where Rq = R{(po) is the radius of curvature of a at (/Jo- 
Let I = Ii n I2 and a be a normal perturbation a{f) = ^(v^) + X{f)ri{ip) where \{(p) = if 
(f ^ I, A(v9o) = X'ifo) = 0, A"(v9o) 7^ and ||A||2 is sufficiently small in order that a e Un and 
Ta and are C^-close. The two curves a and a have a contact of order 1 at ipo and the radius 
of curvature of a at this contact point is Rq = Rq ~ A"((y9o). As a and a differs only on / \ {v^o}, 
{(^Q, 6*0), (^n_i, 9n-i)} is also a ra-periodic hyperbolic orbit for T^. Moreover, every trajectory 
not hitting a{I) is the same both for a and for a. 

Let Ty*(^05^o) and W^(j^j,9j) be the stable and unstable curves of (Jpq,9o) and (jpj,9j) under 
Tq. We can choose ||A||2 sufficiently small such that both are also local graphs over / given by 
9 = 9^{%l)) and 9 = 9''^{ip), where ip = "^(v^) is the angular parameter of a with <^o = '4'ifo)- 
The pencil of rays defined by {ilj,9'^{ip)) will focuses backward at the distance and the pencil 
{ip , 9^ (ip)) will focuses forward at the distance c?^ with 

^ (i?o-A"(y;o))singo _ (i?o - X"i^o)) sin^o 



As the contact of a and a on y^o is of order 1, preserving the point and the tangent, the trajectory 
of {lpq,9o) is also the same for both billiards. Then {(po,9o) is a heteroclinic point for Ta and 
9o = nipo) = 9'iVo). 

As the curve is unchanged outside /, the beam of trajectories given by {ip,9^{ilj)) remains the 
same until it hits a{I), implying that = d- and then 

Applying the same construction for the stable curves , with and T^^ gives rf^ = d^ and 

implies that the invariant curves for will intersect transversally. ■ 

The same reasoning also works for the invariant curves of points on different hyperbolic orbits. 
As we have only a countable number of hyperbolic orbits, each one with a finite number of points, 
we can conclude that: 
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Theorem 2 Generically, for billiards on ovals , the invariant curves of two hyperbolic points are 
transversal. 

We remark that we do not prove that every homo/heteroclinic orbit is transversal. We do know 
that genericaUy two invariant stable and unstable curves either do not intersect or have at least 
one transversal homoclinic orbit, but there can also be tangent orbits. 

3 Generic Dynamics 

3.1 Rotational Invariant Curves 

A closed, simple, continuous curve 7 C [0, 27r) x (0, vr) which is not homotopic to a point is called 
a rotational curve. It is invariant if Tq,(7) = 7. The phase-space of the circular billiard, for 
instance, is foliated by rotational invariant curves. On the other hand, there are billiards on ovals 
with no rotational invariant curves at all, as showed in |8]. However, for sufficiently differentiable 
ovals ([6], [12]), the Twist property implies that Tq, has rotational invariant curves in any small 
neighbourhood of the boundaries Bq = [0, 27r) x {0} and 5^ = [0, 27r) x {vr} of the cylinder 
[0,27r) X (0,7r). 

Proposition 10 For generic oval billiards the rotation number of any rotational invariant curve 
is irrational. 

Proof: A rotational invariant curve 7 is a Lipschitz graph over [0, 27r) So '~f{<f) = {!f,6{(f)), 
where 9 is continuous and there exists L > such that \0{ipi) — 0{(po)\ < L\{pi — ipo], \/ipi,(po G 
[0, 27r). Let Il{(p, 6) = ip and f{ip) = Il{Ta{ip, 0{ip)). Then / is a homeomorphism of the circle and 
so its degree is ±1. 

Suppose that the rotation number of 7 is rational. If deg(/) = 1 then / has periodic orbits, all 
with the same period. If deg(/) = —1 then has periodic orbits, all with the same period. 
As Ta is generic, there is only a finite number of periodic orbits, for each period, and they are 
nondegenerate. Hence there exists an N such that F = f^ is a homeomorphism with a finite 
number of fixed points, all nondegenerate. Let ipi < ip2 be two consecutive fixed points. Then for 
every ip G (ipiyi^^), ^ < F{ip) < ... < F^{ip) < F^~^^{ip) < ... < ip2, F^{ip) ip2 a.?, j ^ 00 and 
F^{(p) ipi as i —00. Let e > be such that ipi < ip2 — ^ and let 77 = 6{(p)),ip2 — ^ < f < 
7^2} andC7 = {iip,e),i;2-e < ^ < i^2, |^-^(^^2)| < L\^-H}- Clearly T^^ (77) C % C C~ . Let 
us suppose that {i/j2, ^'(^'2)) is an elliptic periodic point of T^. Then DT^ {ip2, is a rotation 

of angle p ^ 0,7r,27r. There exists M such that L'T^^(V'2, ^(^2))(C7) n = {(^'2, ^(^^2))}- As 
Tq, is at least then there is eo < e such that {1^2, 0{4'2)) is the unique fixed point of in C~ 
and T*^^(C-) n C- = {(^2,^(^2))}. But this is impossible since T^'^i^-) C C". 
Then all periodic points in 7 must be hyperbolic. But this is also impossible in the generic case 
since 7 will be the union of the periodic points and saddle connections. So, the rotation number 
must be irrational. ■ 

From the proof of the above proposition we have the following results: 

Lemma 11 Generically, rotational invariant curves can not cross the middle segment [0, 27r) x 
{n/2}. 

Proof: Let, as before, H{ip, 9) = tt — 6) be the reversing symmetry. Clearly 7 is a rotational 
invariant curve if and only if -^(7) is also a rotational invariant curve. Suppose that a point 
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((y9o,7r/2) G 7. Then {ipQ,Ti/2) G 7n//(7) and, since they are invariant, Tai^^o.TX /2) = 7r/2) G 
7 n -^(7) and {((y9o, vr/2), 7r/2)} is a 2-periodic orbit in 7, which is impossible. Then, for 
a generic oval billiard, if (y9o,^o) ^ 7 and < 7r/2 (resp. > ^/2) then < 7r/2 (resp. 

^(yj) > 7r/2) for all G [0,27r) or, in other words, the orbits on 7 respect the order of the circle 
[0, 27r) (respec. reverse the order). ■ 

Lemma 12 Let be a rotational invariant curve and 0{p) be a hyperbolic periodic orbit, with 
unstable manifold and stable manifold . Then fl 7 = and ("17 = 0. 

Proof: Suppose that x G fl 7. Then T^x 0{j>) and then 0{j)) C 7, since 7 is a continuous 
invariant curve. But 7 has irrational rotation number and then do not contain any periodic orbit. 
The argument for is analogous. ■ 



3.2 Instability regions 

Birkhoff [3] called the region between two invariant rotational curves, with no other invariant 
rotational curves inside, an instability region. In this subsection we will characterize the instability 
regions for a generic oval billiard. 

Definition 13 A cylinder R C [0,27r) x [0,7r] is a non-empty closed connected set such that 
n(i?) = [0, 27r) and whose boundaries are two continuous rotational curves ri and r2- 

Note that we are not asking neither the cylinder nor the boundary curves to be invariant under 
T 

-I. Q,. 

Theorem 3 Let be generic and 0{p) be a hyperbolic periodic orbit such that its unstable 
manifold satisfies Il{W^) = [0, 2tt). Then the smallest cylinder containing is an instability 
region. 

Before proving this theorem, let us remark that for a generic oval billiard and for each period n, 
there is only a finite number of n-periodic hyperbolic orbits and at least one of them is a global 
minimizer of the action Gm,n, for a suitable m. This implies that it has heteroclinic connections 
[1] and then 11 (VT") = [0,27r). Hence the projection hypothesis applies to at least one orbit of 
each period. 

Proof: Let p = (jpo, 9q) be a hyperbolic periodic point, 0{p) its orbit and the unstable 
manifold of 0{p), with YliW) = [0,27r). 

First, assume that there is an invariant rotational curve 7(^5) = {(p,6{(f>)) such that 0{p) lies in 



the cylinder bounded by 7 and -^(7). Since, by lemma 11, 7 has irrational rotation number 
we have that p ^ 7 and fl 7 = fl (7) = 0. Moreover, is invariant and contains W^. 

Let then, C C be the smallest cylinder containing W^. Since T„C is also a cylinder and 
W C C, TgW = W C T^C, and we have that C T„C. So, T„C is also a cylinder 
containing W^. and as C is the smallest cylinder, we must have C C T^C. However, T^ is 
area-preserving, so we conclude that TaC = C, i.e, C is an invariant cylinder. 
As Ta is a diffeomorphism, T^dC = dC and Taint{C) = int{C). From Birkhoff 's Theorem it 
follows that dC is the union of two rotational invariant curves ci and 02- 



In fact, these two curves Ci and C2 are contained in W'^. To prove this fact, let us suppose that 
there is a point p E ci,p ^ W^. Then there is an open ball S, centered at p such that BnW^ = 0. 
We can then construct a new cylinder C C C with dC = {£1,02} where ci = ci outside B and 
Ci = dB n C otherwise. 
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We have then a new cylinder satisfying W'^ C C C C and so C is not the smallest one, which is a 
contradiction. Now, as is the union of continuous curves, its closure contains the boundaries 
ci and C2 and W'^D'y = for every 7 rotational and invariant, it follows that there is no invariant 
curve in int{C). 

We have proved that C is an invariant cylinder, its boundaries are rotational invariant curves and 
there are no other rotational invariant curves inside it. So, it is the instability region containing 

Suppose now that there is no invariant rotational curves curves bounding from bellow, i.e, 
< 6*0 < 0{^o) < TT — 9{(po) for all rotational invariant curves 7(v5) = {(p,9{ipj). We can extend 
Ta continuously to the circle Bq = [0,27r) x {0}, observing that Ta{Bo) = Bq. Now, any cylinder 
with boundaries Bq and a rotational invariant curve contains W'". The smallest one, C, will have 
boundaries Bq and C2, because if there were a new boundary ci then it will be a rotational invariant 
curve with 0{(Pq) < < tt — 9{ipo). Reasoning as above we conclude that C is the instability 
region that contains W^. 

The case 9{lpq) < tt — 6{(po) < 9q for all rotational invariant curves follows from the reversibility 
property of T^. 

Finally, if has no rotational invariant curves, the instability region containing is the entire 
cylinder [0,27r) X [0,7r]. ■ 

Obviously, we can make a similar construction and obtain the same result for the stable manifold 



3.3 Islands of Stability 

Proposition 14 Let V he the complement ofW^ in the instability region C. Then V = UfUi, 
where all the Ui 's are disjoint open sets, homeomorphic to discs. Dynamically, each Ui returns 
over itself. 

Proof: V is open and is the union of its disjoint open connected components. Let U be one of 
them. U can not divide C into two disjoint sets because contains H^", which projects over 
[0, 27r) and converges to the two rotational curves of dC. So U is homotopically trivial. 

As y C C C [0, 27r) X [0, tt] has finite area and each connected component has positive area. So 
there can not exist a non countable number of them. 

Clearly, TaV — V. As Tq is area preserving then, given Ui, there exists a smallest n{i) such that 
Ta^^Ui n f/i ^ 0. But, as each Ui is a connected component V, we must have Ta^^^Ui — Ui. In 
other words, each component Ui of V = C — is periodic. ■ 

The above proposition suggests the definition 

Definition 15 Given i, let n{i) be the smallest integer such that Ta^^^Ui — Ui. We call Ui an 
island of stability andn{i) its period. The invariant set I — UiUT^UiU ...UTa^^'' ^Ui will be called 
an archipel. 

Remark that, as each island Ui is invariant by Ta '"'^ and homeomorphic to a ball then 
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Proposition 16 The boundary dUi is a closed connected set, with empty interior, contained in 
and invariant by Ta^^'' . 



An easy consequence of proposition 14 



IS 



Proposition 17 Every periodic point in an island of stability has a period multiple of the period 
of the island. 

More strongly, we have that 

Proposition 18 Each island of stability contains a periodic point with the same period as the 
island. 

Proof: Let Ui be an island of stability with period n{i). Since the restriction Ta^^''\Ui is area 
preserving and conjugated to a diffeomorphism of the plane which is orientation preserving, it 
follows from Brouwer's Translation Theorem that Ta^*^|?7j has a fixed point. So there exists a 
point Pi G Ui such that Ta^'^^pi = pi and so a periodic orbit 0{pi) = {pi,TaPi, ...,Ta*-*^ ^pi} with 
the smallest possible period. ■ 

Remark that, as billiards have no fixed points then T^Ui fl t/j = and n{i) > 2. 

Proposition 19 All points in an island of stability have the same rational rotation type. 

Proof: Let, as above, Ui be an island of stability with period n{i). Following [13] we observe that 
if Ui is the lift of Ui to the universal cover and Tq, is a lift of Tq,, then there exists an integer m 

such that Ta ^ \Ui) = Ui + (2m7r, 0). Since Ui is bounded, this implies that that for all z G Ui ii 

kn(i) 

z denotes a lift of z, then there exists lim = — -— , where pri is the projetion on 

n^oo k n[i) 

the first factor. ■ 



3.4 The Instability Set 

For the sake of completeness, we conclude by describing some well known dynamical consequences 
of the area preserving and the twist properties. 

Let Tq, be generic, in the sense of the previous sections, and 0{p) be a hyperbolic periodic orbit 
such that its unstable manifold W satisfies n(H^*') = [0,27r). Let C be the instability region 
containing W^. Following the terminology of Franks in [7] we call W'^ the instability set. 

Clearly, the instability set W"^ does not contain any Moser stable periodic orbit, since they have 
open islands around them. For hyperbolic periodic orbits we have 



Proposition 20 Let 0{q) be a hyperbolic periodic orbit in and let Vr^(g) and W'^{q) be its 
invariant stable and unstable manifolds. Then either W^{q) and W'^{q) C or q E dUi, ie, the 
boundary of an island. Moreover, for j = u or s, ifW^{q) C W^" and 'n.{W^{q)) = [0, 27r) then 
W^{q) = W^; or if q E dUi then two branches of the invariant curves ofW^{q) and/or W^{q) are 
contained in dU. 



Proof: Suppose that there exists z q such that z G W^{q) fl W^, j = u, s. Then by lemma 3.1 
of [13], the branch of W^{q) which contains z is entirely contained in W^. So if each branch of 
W^{q) intersects then W^{q) C W" and obviously, W^{q) C W^. 
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Let us now suppose that a branch 7 of W^{q) is not in W"^. Then 7 is contained in an island of 
stabihty f/j or is outside the region of instabihty C. As the u, a-hmit of 7 is q, either q G dUi or 
g G Ci or C2, the boundary curves of C. But this last case is impossible in the generic case. So 
q G dUi. But in this case, it is enough to remark that dUi is an invariant closed connected set and 
that the only sets with these properties containing q are the invariant manifolds to conclude that 
two branches of the invariant curves of W^{q) and/or W^{q) are contained in dUi. 

If U{W^{q)) = [0, 27r) then q ^ dUi and, by theorem § W^{q) =W^. ■ 



Concerning non-periodic orbits and as a consequence of proposition 19 we have that 



Proposition 21 Any Aubry- Mather set of irrational type contained in C is actually contained 
in the instability set W^. 



And as it was proven by Le Calvez |T3] The instability set W"^ contains the closure of the orbits 
which a and/or cu-limits are ci and 02- 



4 Final remarks 

So far we have obtained a rough description of the dynamics in the instability region, very similar 
to what hapens for generic area preserving twist maps. It is made of the closure of an hyperbolic 
orbit and the union of periodic islands. 

In order to proceed with this description it is necessary to address two basic related themes. The 
first one has to do with the islands: the existence of a finite number of them, the existence of 
a lower bound for the period of an island (e.g. there are examples with no period two islands) 
and the existence of connecting orbits between the boundaries of the islands. A very interesting 
question is if it is possible to have an instability region without any islands. 

The other concerns the properties of W'^: does it have empty interior? Is it topologically transitive? 

A starting point is the instability region that contains the period two orbits. In this region, the 
reversibility symmetry of the phase space, together with geometric properties of the boundary of 
the billiard may allow us to obtain new results about the above questions. 
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